0.VERVIEW

During the second week of each unit, we’ll “walk through” a basic research workflow, or data analysis process, modeled after the Data-Intensive Research Workflow from Learning Analytics Goes to School (Krumm et al., 2018):

Figure 2.2 Steps of Data-Intensive Research Workflow

Each walkthrough will focus on a basic analysis using text mining techniques that you’ll be expected to reproduce, and apply to a new research question next week, using the provided dataset or dataset of your own choosing.

This week, we will focus on analysis of open-ended survey items from an evaluation of the North Carolina Department of Public Instruction (NCDPI) online professional development offered as part of the state’s Race to the Top efforts. For more information about the Race to the Top evaluation work, visit https://cerenc.org/rttt-evaluation/.

Walkthrough Focus

For Unit 1, our focus will be on getting our text “tidy” so we can perform some basic word counts, look at words that occur at a higher rate in a group of documents, and examine words that are unique to those document groups. Specifically, the Unit 1 Walkthrough will cover the following workflow topics:

  1. Prepare: Prior to analysis, it’s critical to understand the context and data sources you’re working with so you can formulate useful and answerable questions. You’ll also need to set up a “Project” for our Unit 1 walkthrough.
  2. Wrangle: Wrangling data entails the work of manipulating, cleaning, transforming, and merging data. In section 2 we focus on reading, reducing, and tidying our data.
  3. Explore: In section 3, we use simple summary statistics, more sophisticated approaches like term frequency-inverse document frequency (tf-idf), and basic data visualization to explore our data and see what insight it provides in response to our question.

While we won’t investigate approaches to Model our data until Unit 3 when we learn about topic models, we will be developing “data products” next week to Communicate our findings and insights.

1. PREPARE

Prior to analysis, it’s critical to understand the context and data sources available so you can formulate useful questions that can be feasibly addressed by your data. For this section, we’ll focus on the following topics:

  1. Context. We learn a little bit about the North Carolina Race to the Top grant and the evaluation of the online professional development work, including questions and findings from the 2012 report.
  2. Questions What aspects of online professional development offerings do teachers find most valuable?
  3. Project Setup We revisit Chapter 6 from DSIEUR to set up a Project in RStudio and ready and R script for analysis.

a. Little Context

RTT Online Professional Development Evaluation

2012 Annual Report

North Carolina was one of 12 recipients of the 2010 federal Race to the Top (RttT) grants, bringing nearly $400 million to the state’s public school system. Over the course of four years, NC’s RttT coordinated a set of activities and policy reforms designed to collectively improve the performances of students, teachers, leaders, and schools.

The North Carolina Race to the Top (RttT) proposal (North Carolina Office of the Governor, 2010) specifies that the state’s Professional Development Initiative will focus on the “use of e- learning tools to meet the professional development needs of teachers, schools, and districts” (p. 191). It points to research demonstrating that “well-designed and -implemented online professional development programs are not only valued by teachers but also positively impact classroom practices and student learning.”

Data Source & Analysis

The evaluation used a wide range of data sources including interviews, document review, site analytics, and surveys, which we’ll focus on for this walkthrough. Survey protocols were designed in cooperation with NCDPI to systematically collect information about local professional development, state-level supports, use of available RttT professional development resources, and organizational and classroom practices in the schools, which will serve as a baseline to assess changes over the period of the North Carolina RttT initiatives.

Quantitative analyses focused primarily on descriptive analysis of item-level responses. In addition, quantitative data from these surveys were analyzed to examine patterns in responses by participants’ role, event type (e.g., module, webinar, resource), and region. Responses to open-ended survey items of the Online Resources Survey were manually coded by their relation to each Learning Forward professional development standard.

Note that the dataset we’ll be using for analysis in this walkthrough is exported as is from Qualtrics with personal identifiers, select demographics, metadata, and closed-ended responses removed.

Summary of Findings

Approximately half of the state’s educators completed at least one online module by the end of the 2011-12 school year. Overall, most participants agreed that the webinars and modules were relevant to their professional development needs, though some content was redundant with prior PD activities and not always content- or grade-specific, and some modules did not meet national standards. Most online modules were completed independently and not in Professional Learning Community groups.

A common theme from focus groups and open-ended survey responses was the convenience of online professional development. One teacher in a focus group stated, “I liked the format. And the way that it was given, it was at your own pace, which works well for our schedules…” Educators also frequently cited that the information and resources provided through the modules improved their understanding of the new standards and the teacher evaluation process. Webinar participants appreciated the useful, updated information presented through a combination of PowerPoint slides and video clips.

While the majority of educators have indicated their satisfaction with these resources, the findings suggest that the use of these resources at both the state and local level was not wholly consistent with national standards for online professional development. Many LEAs likely needed additional guidance, training, support, technology tools, and/or content resources to ensure that local efforts contribute to the quality of the experiences for educators and that the vision for online professional development outlined in the state’s RttT proposal is realized and can be sustained beyond RttT.

b. Guiding Questions

The State’s progress on designing and implementing online professional development was originally guided by the following (very) general evaluation questions:

  1. State Strategies: To what extent did the state implement and support proposed RttT professional development efforts?
  2. Short-Term Outcomes: What were direct outcomes of state-level RttT professional development efforts?

For this walkthrough, we’ll use text mining to complement prior qualitative analyses conducted as part of the RttT Evaluation by examining responses to open-ended questions on the RttT Online PD Survey administered to over 15,000 NC educators.

Our (very) specific questions of interest for this walkthrough are:

  1. What aspects of online professional development offerings do teachers find most valuable?
  2. How might resources differ in the value they afford teachers?

Finally, one overarching question we’ll explore throughout this course, and that Silge and Robinson (2018) identify as a central question to text mining and natural language processing, is:

How do we to quantify what a document or collection of documents is about?

c. Set Up a Project

As highlighted in Chapter 6 of Data Science in Education Using R (DSIEUR), one of the first steps of every workflow should be to set up a “Project” within RStudio. This will be your “home” for any files and code used or created in Unit 1. Open RStudio and follow these steps from DESIUR 6.6 to create a Project for Unit 1:

  1. Click on “File”
  2. Select “New Project”
  3. Choose “New Directory”
  4. Click on “New Project”
  5. Enter your Project’s name in the box that says, “Directory name”. Choose a Project name that helps you remember that this is a project is part of Unit 1: Tidy Text & Word Counts. Avoid using spaces in your Project name, and instead, separate words with hyphens or underscore characters.
  6. Choose where to save your Project by clicking on “Browse” next to the box labeled “Create project as a subdirectory of:”. If you are just using this to learn and test out creating a Project, consider placing it in your downloads or another temporary directory so that you remember to remove it later.
  7. Click “Create Project”

Create New File

Now that you have a Project to store .R scripts that you create as you work through this unit, let’s create our first .R script:

  1. Click on “File”
  2. Select “R script”
  3. Click the disk icon or select “Save” from file menu to save.
  4. Enter a file name. See the R Style Guide for standard file naming conventions.

Finally, using your newly created R script, type the following code to load the packages we installed last week and that we’ll be needing for this walkthrough.

library(tidyverse)
## ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.0 ──
## ✓ ggplot2 3.3.3     ✓ purrr   0.3.4
## ✓ tibble  3.0.5     ✓ dplyr   1.0.2
## ✓ tidyr   1.1.2     ✓ stringr 1.4.0
## ✓ readr   1.4.0     ✓ forcats 0.5.0
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## x dplyr::filter() masks stats::filter()
## x dplyr::lag()    masks stats::lag()
library(tidytext)

At the end of this week, I’ll ask that you share with me your r script as evidence that you have complete the walkthrough. Although I highly recommend that that you manually type the code shared throughout this walkthrough, for large blocks of text it may be easier to cut and paste.

2. WRANGLE

In general, data wrangling involves some combination of cleaning, reshaping, transforming, and merging data (Wickham & Grolemund, 2017). The importance of data wrangling is difficult to overstate, as it involves the initial steps of going from raw data to a dataset that can be explored and modeled (Krumm et al, 2018).

  1. Reading Data. Before working with data, we need to “read” it into R. It also helps to inspect your data
  2. Data Reduction. We focus on tools from the dplyr package to view, rename, select, slice, and filter our data in preparation for analysis.
  3. Tidy Text. We’ll learn how to use the tidytext package to both “tidy” and tokenize our text in order to create a data frame to use for analysis.

2a. Read, View, Write Data

The Reading Data section introduces the following functions for reading data into R and inspecting it’s contents:

  • dplyr::read_csv() Reading .csv files into R.
  • base::print() View your data frame in the Console Pane
  • utils::view() View your data frame in the Source Pane
  • tibble::glimpse() Like print, but transposed so you can see all columns
  • utils::head() View the first 6 rows of your data.
  • utils::tail() View last 6 rows of your data.

Remember, the name before the double colon indicates the package the function comes from. For example, read_csv comes from the `readr`` package.

Reading Data into R

To get started, we need to import, or “read”, our data into R. The function used to import your data will depend on the file format of the data you are trying to import.

  1. Download the opd_survey.csv file we’ll be using for this Unit from our NCSU Moodle course site.
  2. Create a folder in directory on your computer where you stored your R Project and name it “data”.
  3. Add the file to your data folder.
  4. Check your Files tab in R Studio to verify that your file is indeed in your data folder.

Now let’s read our data into our Environment and assign it to a variable name so we can work with it like any other object in R.

opd_survey <- read_csv("data/opd_survey.csv")
## Warning: Duplicated column names deduplicated: 'Resource' => 'Resource_1' [10],
## 'Resource_10_TEXT' => 'Resource_10_TEXT_1' [11], 'Q16' => 'Q16_1' [12]
## 
## ── Column specification ────────────────────────────────────────────────────────
## cols(
##   RecordedDate = col_character(),
##   ResponseId = col_character(),
##   Role = col_character(),
##   Q14 = col_character(),
##   Q16 = col_character(),
##   Resource = col_character(),
##   Resource_8_TEXT = col_character(),
##   Resource_9_TEXT = col_character(),
##   Resource_10_TEXT = col_character(),
##   Resource_1 = col_character(),
##   Resource_10_TEXT_1 = col_character(),
##   Q16_1 = col_character(),
##   Q16_9_TEXT = col_character(),
##   Q19 = col_character(),
##   Q21 = col_character(),
##   Delta = col_character(),
##   Q26 = col_character(),
##   Q37 = col_character(),
##   Q8 = col_character()
## )

Notice that read_csv() dealt with the issues of duplicate column names for us!!

If you happen to run into issues with data import, RStudio as has an Import Dataset feature for a point and click approach to adding data to your environment. Be sure to pay attention to the

Viewing Data in RStudio

Once your data is in R, there are many different ways you can view it. Give each of the following at try:

# enter the name of your data frame and view directly in console 
opd_survey 

# view your data frame transposed so your can see every column and the first few entries
glimpse(opd_survey) 

# look at just the first six entries
head(opd_survey) 

# or the last six entries
tail(opd_survey) 

# view the names of your variables or columns
names(opd_survey) 

# or view in source pane
view(opd_survey)

Writing Data to your Project Folder

In addition to reading data from your project folder, you can also write data back to a folder. The readr package has an intuitively named write_csv() function for doing just that.

Using the following code to create a copy of the opd_survey.csv file in your data folder from the opd_survey data frame you created:

write_csv(opd_survey, "data/opd_survey_copy.csv")

Note that the first argument is the data frame you created earlier and the second argument is the file name you plan to give it, including (if necessary) the file path for where it should go.

Comprehension Check

Throughout this walkthrough, you will be asked to respond to questions or short tasks to check your comprehension of the content covered. For section 2a. Read, View, and Write Data, please respond to these questions by commenting out a line or lines in your R script like so:

# 1. What argument would you add to `read_csv()` if my file did not not have column names or headers? 

# I would need to add the ____ argument and set it to equal ____ to prevent R from setting the first row as column names.
  1. What argument would you add to read_csv() if my file did not not have column names or headers? You can type ?read_csv to get help on this function or check this handy cheatsheet for the readr package from the readr website at https://readr.tidyverse.org/index.html
  2. What is the first argument that read_csv() always expects and what happens if you don’t include in quotes?
  3. What package would I need if I wanted to read a Microsoft Excel file into R? Hint: Check the cheatsheet.
  4. What are some of the pros and cons of view() compared to other functions for viewing your data?
  5. What happens if you use write_csv(opd_survey, "opd_survey_copy.csv") and just specify the file name instead including the folder?
  6. What immediate data wrangling issues do you anticipate we’ll have to deal with now that you’ve looked at our data?

2b. Reduce Data

As you’ve probably already noticed from viewing our dataset, we clearly have more data than we need to answer our rather basic research question. For this part of our workflow we focus on the following functions from the dplyr package for wrangling our data:

dplyr functions

  • select() picks variables based on their names.
  • slice() lets you select, remove, and duplicate rows.
  • rename() changes the names of individual variables using new_name = old_name syntax
  • filter() picks cases, or rows, based on their values in a specified column.

stats functions

  • na.omit() a handy little function from the stats package for removing rows with missing values, i.e. NA.

Subset Columns

To begin, let’s select() Role, Resources, and Q21 columns and store as new data frame since those respectively pertain to educator role, OPD resource they are evaluating, and, as illustrated by the second row,

opd_selected <- select(opd_survey, Role, Resource, Q21)

Notice that like the bulk of all tidyverse functions, the first input select() expects is a data frame, followed by the columns you’d like to select.

Let’s take a look at our newly created data frame that should have dramatically fewer variables:

head(opd_selected)
## # A tibble: 6 x 3
##   Role                           Resource                         Q21           
##   <chr>                          <chr>                            <chr>         
## 1 "What is your role within you… "Please indicate the online pro… "How are you …
## 2 "{\"ImportId\":\"QID2\"}"      "{\"ImportId\":\"QID3\"}"        "{\"ImportId\…
## 3 "Central Office Staff (e.g. S… "Summer Institute/RESA PowerPoi…  <NA>         
## 4 "Central Office Staff (e.g. S… "Online Learning Module (e.g. C… "Collecting d…
## 5 "School Support Staff (e.g. C… "Online Learning Module (e.g. C…  <NA>         
## 6 "School Support Staff (e.g. C… "Calendar"                       "time analysi…

Rename Columns

Notice that Q21 is not a terribly informative variable name. Let’s now take our opd_selected data frame and use the rename() function along with the = assignment operator introduced last week to change the name from Q21 to “text” and save it as opd_renamed.

This naming is somewhat intentional because not only is it the text we are interested in analyzing, but also mirrors the naming conventions in our [Text Mining with R]https://www.tidytextmining.com/tidytext.html course book and will make it easier to follow the examples there.

opd_renamed <- rename(opd_selected, text = Q21)

Subset Rows

Now let’s deal with the legacy rows that Qualtrics outputs by default, which are effectively 3 sets of headers. We will use the slice() function, which is basically the same as the select() function but for rows instead of columns, to carve out those two rows.

opd_sliced <- slice(opd_renamed, -1, -2) # the - sign indicates to NOT keep rows 1 and 2

head(opd_sliced)
## # A tibble: 6 x 3
##   Role                      Resource                    text                    
##   <chr>                     <chr>                       <chr>                   
## 1 Central Office Staff (e.… Summer Institute/RESA Powe… <NA>                    
## 2 Central Office Staff (e.… Online Learning Module (e.… Collecting district fee…
## 3 School Support Staff (e.… Online Learning Module (e.… <NA>                    
## 4 School Support Staff (e.… Calendar                    time analysis           
## 5 Teacher                   Live Webinar                incorporating into all …
## 6 Teacher                   Online Learning Module (e.… Answering evaluation qu…

Now let’s take our opd_sliced and remove any rows that are missing data, as indicated by an NA.

opd_complete <- na.omit(opd_sliced)

Finally, since we are only interested in the feedback from teachers, let’s also filter our dataset for only participants who indicated their Role as “Teacher”.

opd_teacher <- filter(opd_complete, Role == "Teacher")

head(opd_teacher)
## # A tibble: 6 x 3
##   Role   Resource                             text                              
##   <chr>  <chr>                                <chr>                             
## 1 Teach… Live Webinar                         "incorporating into all content a…
## 2 Teach… Online Learning Module (e.g. Call f… "Answering evaluation questions." 
## 3 Teach… Online Learning Module (e.g. Call f… "Use the online module to reflect…
## 4 Teach… Online Learning Module (e.g. Call f… "Preparing for Common core"       
## 5 Teach… Online Learning Module (e.g. Call f… "gain knowledge"                  
## 6 Teach… Summer Institute/RESA PowerPoint Pr… "informing instructional practice…

That was a lot of code we just wrote to end up with opd_teacher. Let’s review:

opd_selected <- select(opd_survey, Role, Resource, Q21)
opd_renamed <- rename(opd_selected, text = Q21)
opd_sliced <- slice(opd_renamed, -1, -2)
opd_complete <- na.omit(opd_sliced)
opd_teacher <- filter(opd_complete, Role == "Teacher")

Note that we could have reused opd_teacher and simply overwritten it each time to prevent creating new objects:

opd_teacher <- select(opd_survey, Role, Resource, Q21)
opd_teacher <- rename(opd_teacher, text = Q21)
opd_teacher <- slice(opd_teacher, -1, -2)
opd_teacher <- na.omit(opd_teacher)
opd_teacher <- filter(opd_teacher, Role == "Teacher")

Code Reduction

Fortunately, we can use the Pipe Operator %>% introduced in Chapter 6 of Data Science in Education Using R (DSIEUR) to dramatically simplify these cleaning steps and reduce the code written

opd_teacher <- opd_survey %>%
  select(Role, Resource, Q21) %>%
  rename(text = Q21) %>%
  slice(-1, -2) %>%
  na.omit() %>%
  filter(Role == "Teacher")

head(opd_teacher)
## # A tibble: 6 x 3
##   Role   Resource                             text                              
##   <chr>  <chr>                                <chr>                             
## 1 Teach… Live Webinar                         "incorporating into all content a…
## 2 Teach… Online Learning Module (e.g. Call f… "Answering evaluation questions." 
## 3 Teach… Online Learning Module (e.g. Call f… "Use the online module to reflect…
## 4 Teach… Online Learning Module (e.g. Call f… "Preparing for Common core"       
## 5 Teach… Online Learning Module (e.g. Call f… "gain knowledge"                  
## 6 Teach… Summer Institute/RESA PowerPoint Pr… "informing instructional practice…

Our dataset is now ready to be tidied!!!

Comprehension Check

  1. How would you change the code in the Subset Columns section if you wanted to analyze how educators are are using the online resources instead of the most beneficial aspects?
  2. How would you rewrite the code in the Subset Rows section if you were interested in looking at School Executive responses instead of teachers?
  3. Rewrite the piped code in the Code Reduction section to prepare a data frame for tidying that includes all Roles, not just teachers, but excludes the Resource column. Assign it to opd_benefits for later use.

2c. Tidy Text

For this part of our workflow we focus on the following functions from the tidytext and dplyr packages respectively:

  • unnest_tokens() splits a column into tokens
  • anti_join() returns all rows from x without a match in y.

Tidy Data Principles

Not surprisingly, the Tidyverse set of packages including packages like dplyr adhere “tidy” data principles (Wickham 2014). Tidy data has a specific structure:

  • Each variable is a column
  • Each observation is a row
  • Each type of observational unit is a table

Why would this data be considered “untidy”?

Text data, by it’s very nature is ESPECIALLY untidy. In Chapter 1 of Text Mining with R, Silge and Robinson define the tidy text format as

a table with one-token-per-row. A token is a meaningful unit of text, such as a word, that we are interested in using for analysis, and tokenization is the process of splitting text into tokens. This one-token-per-row structure is in contrast to the ways text is often stored in current analyses, perhaps as strings or in a document-term matrix. For tidy text mining, the token that is stored in each row is most often a single word, but can also be an n-gram, sentence, or paragraph. In the tidytext package, we provide functionality to tokenize by commonly used units of text like these and convert to a one-term-per-row format.

In this section, our goals is to transform our opd_teacher data from this:

## # A tibble: 6 x 3
##   Role   Resource                             text                              
##   <chr>  <chr>                                <chr>                             
## 1 Teach… Live Webinar                         "incorporating into all content a…
## 2 Teach… Online Learning Module (e.g. Call f… "Answering evaluation questions." 
## 3 Teach… Online Learning Module (e.g. Call f… "Use the online module to reflect…
## 4 Teach… Online Learning Module (e.g. Call f… "Preparing for Common core"       
## 5 Teach… Online Learning Module (e.g. Call f… "gain knowledge"                  
## 6 Teach… Summer Institute/RESA PowerPoint Pr… "informing instructional practice…

to this:

## # A tibble: 6 x 3
##   Role    Resource     word         
##   <chr>   <chr>        <chr>        
## 1 Teacher Live Webinar incorporating
## 2 Teacher Live Webinar into         
## 3 Teacher Live Webinar all          
## 4 Teacher Live Webinar content      
## 5 Teacher Live Webinar areas        
## 6 Teacher Live Webinar adn

Tokenize Text

In order to tidy our text, we need to break the text into individual tokens (a process called tokenization) and transform it to a tidy data structure. To do this, we use tidytext’s incredibly powerful unnest_tokens() function.

After all the work we did prepping our data, this is going to feel a little anticlimactic.

Let’s go ahead and tidy our text and save it as opd_tidy:

opd_tidy <- unnest_tokens(opd_teacher, word, text)

head(opd_tidy)
## # A tibble: 6 x 3
##   Role    Resource     word         
##   <chr>   <chr>        <chr>        
## 1 Teacher Live Webinar incorporating
## 2 Teacher Live Webinar into         
## 3 Teacher Live Webinar all          
## 4 Teacher Live Webinar content      
## 5 Teacher Live Webinar areas        
## 6 Teacher Live Webinar adn

Note that we also could have just added unnest_tokens(word, text) to our previous piped chain of functions like so:

opd_tidy <- opd_survey %>%
  select(Role, Resource, Q21) %>%
  rename(text = Q21) %>%
  slice(-1, -2) %>%
  na.omit() %>%
  filter(Role == "Teacher") %>%
  unnest_tokens(word, text)

head(opd_tidy)
## # A tibble: 6 x 3
##   Role    Resource     word         
##   <chr>   <chr>        <chr>        
## 1 Teacher Live Webinar incorporating
## 2 Teacher Live Webinar into         
## 3 Teacher Live Webinar all          
## 4 Teacher Live Webinar content      
## 5 Teacher Live Webinar areas        
## 6 Teacher Live Webinar adn

There is A LOT to unpack with this function. First notice that unnest_tokens expects a data frame as the first argument, followed by two column names. The first is an output column name that that doesn’t currently exist but will be created as the text is unnested into it (word, in this case). This if followed by the input column that the text comes from which we uncreatively named text. Also notice:

  • Other columns, such as Role and Resource, are retained.
  • All punctuation has been removed.
  • Tokens have been changed to lowercase, which makes them easier to compare or combine with other datasets. However, we can use the to_lower = FALSE argument to turn off this behavior).

Remove Stop Words

One final step in tidying our text is to remove words that don’t add much value to our analysis (at least when using this approach) such as “and”, “the”, “of”, “to” etc. The tidytext package contains a stop_words dataset derived from three different lexicons that we’ll use to remove rows that match words in this dataset.

Let’s take a look at these common stop words so we know what we’re getting rid of from our opd_tidy dataset.

head(stop_words)
## # A tibble: 6 x 2
##   word      lexicon
##   <chr>     <chr>  
## 1 a         SMART  
## 2 a's       SMART  
## 3 able      SMART  
## 4 about     SMART  
## 5 above     SMART  
## 6 according SMART
view(stop_words)

In order to remove these stop words, we will use function called anti_join() that looks for matching values in a specific column from two datasets and returns rows from the original dataset that have no matches. For a good overview of the different dplyr joins see here: https://medium.com/the-codehub/beginners-guide-to-using-joins-in-r-682fc9b1f119

Let’s remove rows from our opd_tidy data frame that contain matches in the word column with those in the stop_words dataset and save it as opd_clean since we were done cleaning our data at this point.

opd_clean <- anti_join(opd_tidy, stop_words)
## Joining, by = "word"
head(opd_clean)
## # A tibble: 6 x 3
##   Role    Resource     word         
##   <chr>   <chr>        <chr>        
## 1 Teacher Live Webinar incorporating
## 2 Teacher Live Webinar content      
## 3 Teacher Live Webinar adn          
## 4 Teacher Live Webinar students     
## 5 Teacher Live Webinar move         
## 6 Teacher Live Webinar create

Comprehension Check

  1. How would you include the anti_join() function in our previous chain that uses the pipe operator? Give it a try and see what happens.
  2. Why do you think the console provided the message “Joining, by = ‘word’”?
  3. How would you use anti_join() if we had named the output column from unnest_tokens() “tokens” instead? Hint: Check ?anti_join documentation.
  4. Tidy your opd_benefits data by tokenizing your text and removing stop words.

3. EXPLORE

As highlighted in both DSEIUR and Learning Analytics Goes to School, calculating summary statistics, data visualization, and feature engineering (the process of creating new variables from a dataset) are a key part of exploratory data analysis. One goal in this phase is explore questions that drove the original analysis and develop new questions and hypotheses to test in later stages. In Section 3, we will calculate some very basic summary statistics from our tidied text, explore key words of interest to gather additional context, and use data visualization to identify patterns and trends that may not be obvious from our tables and numerical summaries. Topics addressed in Section 3 include:

  1. Summary Stats. We focus primarily on the use of word counts and calculating proportions to to help us identify common words used to describe the most valuable aspects of online professional development offerings.
  2. Word Search. We learn about the global regular expression parser, or grep package in R, to search for key words among our data set.
  3. Data Visualization. Finally, we wrap up this walkthrough and preview our work for next week by creating wordclouds, bar plots, and small multiple charts to explore patterns and trends that would be difficult to distinguish otherwise.

3a. Summary Stats

Prior to making any data visualization, we revisit our or overarching question guiding most of our efforts in this course, “How do we quantify what a text is about?”

In this section, we introduce the following functions:

dplyr functions - count() lets you quickly count the unique values of one or more variables - group_by() takes a data frame and one or more variables to group by - summarise() - mutate() adds new variables and preserves existing ones - left_join() add columns from one dataset to another

tidytext functions - bind_tf_idf() binds the term frequency and inverse document frequency of a tidy text dataset to the dataset

Word Counts

As highlighted in Word Counts are Amazing, one simple but powerful approach to text analysis is counting the frequency in which words occur in a given collection of documents, or corpus.

Now that we have our original survey data in a tidy text format, we can use the count() function from the dplyr package to find the most common words used by teachers when asked, “What was the most beneficial/valuable aspect of this online resource?”

opd_counts <- count(opd_clean, word, sort = TRUE)

# alternatively, we could have use the %>% operator to yield the same result.

opd_counts <- opd_clean %>% 
  count(word, sort = TRUE)

opd_counts
## # A tibble: 2,268 x 2
##    word             n
##    <chr>        <int>
##  1 standards      855
##  2 classroom      793
##  3 core           740
##  4 common         707
##  5 development    590
##  6 teaching       548
##  7 students       512
##  8 professional   470
##  9 resource       424
## 10 understand     382
## # … with 2,258 more rows

Going back to findings from the original report, a strategy as simple basic word counts resulted in key words consistent with findings from the qualitative analysis of focus-group transcripts and open-ended survey responses:

Educators frequently cited that the information and resources provided through the modules improved their understanding of the new standards and the teacher evaluation process.

See also this finding around video clips:

Webinar participants appreciated the useful, updated information presented through a combination of PowerPoint slides and video clips.

One notable distinction between word counts and more traditional qualitative analysis is that broader themes like “convenience” often are not immediately apparent in words counts, but rather emerges from responses containing words like “pace”, “format”, “online”, “ease”, and “access”.

A common theme from focus groups and open-ended survey responses was the convenience of online professional development. One teacher in a focus group stated, “I liked the format. And the way that it was given, it was at your own pace, which works well for our schedules…”

The count() function can also be used with more than one column to count the frequency a word occurs for a select Resource in our dataset.

opd_resource_counts <- opd_clean %>%
  count(Resource, word, sort = TRUE)

view(opd_resource_counts)

In this case, we see that “information” was the most common word for Online Learning Modules but did not even make the top 5 for Recorded Webinar:

Word Frequencies

One common approach to facilitate comparison across documents or groups of text, in our case responses by Online Resource type, is by looking at the frequency that each word occurs among all words for that document group. This also helps to better gauge how prominent the same word is across different groups.

For example, let’s create counts for each Resource and word paring, and then create a new column using the mutate() function that calculations the proportion that word makes up among all words.

To do this a little more efficiently, I’m going to use the %>% operator:

opd_frequencies <- opd_clean %>%
  count(Resource, word, sort = TRUE) %>%
  group_by(Resource) %>%
  mutate(proportion = n / sum(n))

opd_frequencies
## # A tibble: 3,808 x 4
## # Groups:   Resource [10]
##    Resource                                           word          n proportion
##    <chr>                                              <chr>     <int>      <dbl>
##  1 Online Learning Module (e.g. Call for Change, Und… standards   777     0.0378
##  2 Online Learning Module (e.g. Call for Change, Und… classroom   665     0.0323
##  3 Online Learning Module (e.g. Call for Change, Und… core        662     0.0322
##  4 Online Learning Module (e.g. Call for Change, Und… common      585     0.0285
##  5 Online Learning Module (e.g. Call for Change, Und… developm…   513     0.0250
##  6 Online Learning Module (e.g. Call for Change, Und… teaching    484     0.0235
##  7 Online Learning Module (e.g. Call for Change, Und… professi…   412     0.0200
##  8 Online Learning Module (e.g. Call for Change, Und… students    362     0.0176
##  9 Online Learning Module (e.g. Call for Change, Und… resource    346     0.0168
## 10 Online Learning Module (e.g. Call for Change, Und… understa…   336     0.0163
## # … with 3,798 more rows

Using the view() function we can see that “information” makes up about 2.3% of words in responses about the Online Modules, and about 1.7% for Recorded Webinars.

Term Frequency-Inverse Document Frequency

Term frequency-inverse document frequency (tf-idf) is an approach that takes this approach one step further.

As noted in Tidy Text Mining with R:

The statistic tf-idf is intended to measure how important a word is to a document in a collection (or corpus) of documents, for example, to one novel in a collection of novels or to one website in a collection of websites.

Silge and Robinson note that, “The idea of tf-idf is to find the important words for the content of each document by decreasing the weight for commonly used words and increasing the weight for words that are not used very much in a collection or corpus of document… That is, tf-idf attempts to find the words that are important (i.e., common) in a text, but not too common.”

The tidytext package has a function called bind_tf_idf() that takes a tidy text dataset as input with one row per token (term), per document. One column (word here) contains the terms/tokens, one column contains the documents (book in this case), and the last necessary column contains the counts, how many times each document contains each term (n in this example).

Because tf-idf can account through weighting for “too common” words like “and” or “but”, when calculating tf-idf it is not necessary to remove stop words. However, we will need add a column for total words for each Resource type which can be accomplished in a couples steps.

First, let’s recycle our opd_teacher data frame and calculate counts for each word again, but this time instead of word counts for the total data set, we’ll calculate word counts for each ‘Resource’.

opd_words <- opd_teacher %>%
  unnest_tokens(word, text) %>%
  count(Resource, word, sort = TRUE)

head(opd_words)
## # A tibble: 6 x 3
##   Resource                                                           word      n
##   <chr>                                                              <chr> <int>
## 1 Online Learning Module (e.g. Call for Change, Understanding the S… to     3336
## 2 Online Learning Module (e.g. Call for Change, Understanding the S… the    2272
## 3 Online Learning Module (e.g. Call for Change, Understanding the S… my     1555
## 4 Online Learning Module (e.g. Call for Change, Understanding the S… in     1374
## 5 Online Learning Module (e.g. Call for Change, Understanding the S… i      1175
## 6 Online Learning Module (e.g. Call for Change, Understanding the S… and    1129

Next, let’s calculate the total words per Resource type:

total_words <- opd_words %>%
  group_by(Resource) %>%
  summarise(total = sum(n))
## `summarise()` ungrouping output (override with `.groups` argument)
total_words
## # A tibble: 10 x 2
##    Resource                                                                total
##    <chr>                                                                   <int>
##  1 Calendar                                                                  171
##  2 Document, please specify (i.e. Facilitator's Guide, Crosswalks, Sample…   521
##  3 Live Webinar                                                               81
##  4 Online Learning Module (e.g. Call for Change, Understanding the Standa… 47196
##  5 Other, please specify                                                    3773
##  6 Promotional Video                                                         233
##  7 Recorded Webinar or Presentation (e.g. Strategic Staffing, Standards a…  1212
##  8 Summer Institute/RESA PowerPoint Presentations                           1173
##  9 Website, please specify                                                  2139
## 10 Wiki                                                                     1244

Now let’s append the total column from total_words to our opd_words data frame:

opd_totals <- left_join(opd_words, total_words)
## Joining, by = "Resource"
opd_totals
## # A tibble: 5,063 x 4
##    Resource                                                  word        n total
##    <chr>                                                     <chr>   <int> <int>
##  1 Online Learning Module (e.g. Call for Change, Understand… to       3336 47196
##  2 Online Learning Module (e.g. Call for Change, Understand… the      2272 47196
##  3 Online Learning Module (e.g. Call for Change, Understand… my       1555 47196
##  4 Online Learning Module (e.g. Call for Change, Understand… in       1374 47196
##  5 Online Learning Module (e.g. Call for Change, Understand… i        1175 47196
##  6 Online Learning Module (e.g. Call for Change, Understand… and      1129 47196
##  7 Online Learning Module (e.g. Call for Change, Understand… for      1099 47196
##  8 Online Learning Module (e.g. Call for Change, Understand… of        859 47196
##  9 Online Learning Module (e.g. Call for Change, Understand… a         795 47196
## 10 Online Learning Module (e.g. Call for Change, Understand… standa…   777 47196
## # … with 5,053 more rows

Finally, we’re ready to use the bind_tf_idf() function to calculate a tf-idf statistic for each word and assess it’s relative importance to a given online resource type:

opd_tf_idf <- opd_totals %>%
  bind_tf_idf(word, Resource, n)

opd_tf_idf
## # A tibble: 5,063 x 7
##    Resource                              word       n total     tf   idf  tf_idf
##    <chr>                                 <chr>  <int> <int>  <dbl> <dbl>   <dbl>
##  1 Online Learning Module (e.g. Call fo… to      3336 47196 0.0707 0     0      
##  2 Online Learning Module (e.g. Call fo… the     2272 47196 0.0481 0     0      
##  3 Online Learning Module (e.g. Call fo… my      1555 47196 0.0329 0     0      
##  4 Online Learning Module (e.g. Call fo… in      1374 47196 0.0291 0     0      
##  5 Online Learning Module (e.g. Call fo… i       1175 47196 0.0249 0     0      
##  6 Online Learning Module (e.g. Call fo… and     1129 47196 0.0239 0     0      
##  7 Online Learning Module (e.g. Call fo… for     1099 47196 0.0233 0     0      
##  8 Online Learning Module (e.g. Call fo… of       859 47196 0.0182 0     0      
##  9 Online Learning Module (e.g. Call fo… a        795 47196 0.0168 0     0      
## 10 Online Learning Module (e.g. Call fo… stand…   777 47196 0.0165 0.105 0.00173
## # … with 5,053 more rows
view(opd_tf_idf)

Notice that idf and thus tf-idf are zero for these extremely common words (typically stop words). These are all words that appear in teacher responses for all online resource types, so the idf term (which will then be the natural log of 1) is zero. The inverse document frequency (and thus tf-idf) is very low (near zero) for words that occur in many of the documents in a collection; this is how this approach decreases the weight for common words. The inverse document frequency will be a higher number for words that occur in fewer of the documents in the collection.

On one final note, while it has proved useful in text mining, search engines, etc., its theoretical foundations are considered less than firm by information theory experts…"

In the next section, we’ll use some data visualization strategies to help us interpret and find patterns in these rather dense output tables.

Comprehension Questions

  1. Looking back at the Word Counts section, what other aspects of the online professional development resources to our word counts suggest teachers find valuable or beneficial?
  2. Instead of using the view() function for opd_resource_counts and searching in the source how, how might you use the filter() function to find return the most common words for Recorded Webinars?
  3. How many total resources were actually evaluated and which resource received the most feedback? How do you know?
  4. What are some obvious limitations to tf-idf, at least for this dataset, based on the initial opd_tf_idf data frame we created?
  5. Calculate word counts, frequencies, and tf-idf for you opd_benefits data frame. For frequencies and tf-idf, group by Role instead of Resource.
  6. What differences, if any, do you notice between teachers and other roles?

3c. Data Visualization

We covered data visualization pretty extensively in ECI 586: Introduction to Learning Analytics, but for those new to data visualization in R, the go to package for standard charts and graphs is ggplot2. Hadley Wickham’s R for Data Science and [ggplot2: Elegant Graphics for Data] are also great introductions to data visualization in R with ggplot2.

Word Clouds

The wordcloud2 packages is pretty dead simple for generating HTML based word clouds.

For example, let’s load our installed wordclouds2 library, and run the wordcloud2() function on our opd_counts data frame:

library(wordcloud2)

wordcloud2(opd_counts)

I use wordclouds pretty sparingly in evaluation reports, but typically include them for open ended items in online Qualtrics survey reports to provide education partners I work with a quick snapshot of the response.

Once installed, I recommend using ?wordclouds2 to view the various arguments for cleaning up the default view.

Basic Bar Chart

The bar chart is the workhorse for data viz and is pretty effective for comparing two or more values. Given the unique aspect of our tidy text data frame, however, we are looking at upwards of over 5,000 values (i.e. words and their counts) to compare with our opd_counts data frame and will need some way to limit the number of words to display.

opd_counts %>%
  filter(n > 500) %>% # keep rows with word counts greater than 500
  mutate(word = reorder(word, n)) %>% #reorder the word variable by n and replace with new variable called word
  ggplot(aes(n, word)) + # create a plot with n on x axis and word on y axis
  geom_col() # make it a bar plot

#### Small Multiples

Word clouds and bar charts are pretty effective for highlighting the most common words in an entire corpus, or in our case, all teacher survey responses, regarless of resource type being reviewed.

One limitation we ran into earlier when we started looking at word frequencies and tf-idf stats was that it was difficult to easily compare the most common or unique words for each resource type. That is where small multiples come. A small multiple is basically a series of similar graphs or charts using the same scale and axes that make it easier to compare across different document collections of interest, in our case, word counts by resource type.

Let’s use the example illustrated in Text Mining with R to create a small multiple for our opd_frequencies data set instead of the opd_tf_idf

library(forcats)

opd_frequencies %>%
  filter(Resource != "Calendar") %>% # remove Calendar responses, too few. 
  group_by(Resource) %>%
  slice_max(proportion, n = 5) %>%
  ungroup() %>%
  ggplot(aes(proportion, fct_reorder(word, proportion), fill = Resource)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~Resource, ncol = 3, scales = "free")

Comprehension Check

  1. Create a word cloud, bar chart, and/or small multiple using your opd_benefits data.